03. Types of Experiment

Types of Experiment

Most of the time, when you think of an experiment, you think of a
between-subjects experiment. In a between-subjects experiment, each unit
only participates in, or sees, one of the conditions being used in the
experiment. The simplest of these has just two groups or conditions to compare.
In one group, we have either no manipulation, or maintenance of the status quo.
This is like providing a known drug treatment, or an old version of a
website. This is known as the control group. The other group includes the
manipulation we wish to test, such as a new drug or new website layout. This is
known as our experimental group. We can compare the outcomes between groups
(e.g. recovery time or click-through rate) in order to make a judgement about
the effect of our manipulation. (Since we have an experiment, we'll randomly
assign each unit to either the control or experimental group.) For web-based
experiments, this kind of basic experiment design is called an A/B test:
the "A" group representing the old control, and "B" representing the new
experimental change.

We aren't limited to just two groups. We could have multiple experimental
groups to compare, rather than just one control group and one experimental
group. This could form an A/B/C test for a web-based experiment, with control
group "A" and experimental groups "B" and "C".

If an individual completes all conditions, rather than just one, this is known
as a within-subjects design. Within-subjects designs are also known as
repeated measures designs. By measuring an individual's output in all
conditions, we know that the distribution of features in the groups will be
equivalent. We can account for individuals' aptitudes or inclinations in our
analysis. For example, if an individual rates three different color palettes
for a product, we can know if a high rating for one palette is particularly
good compared to the others (e.g. 10 vs. 5, 6) or if it's not a major
distinction (e.g. 10 vs. 8, 9).

Randomization still has a part in the within-subjects design in the order in
which individuals complete conditions. This is important to reduce potential
bias effects, as will be discussed later in the lesson. One other downside of
the within-subjects design is that it's not always possible to pull off a
within-subjects design. For example, when a user visits a website and completes
their session, we usually can't guarantee when they'll come back. The purpose
of their following visit also might not be comparable to their first. It can
take a lot more effort in control in order to set up an effective
within-subjects design.

Side Note: Factorial Designs

Factorial designs manipulate the value of multiple features of interest. For
example, with two independent manipulations "X" and "Y", we have four
conditions: "control", "X only", "Y only", "X and Y". Experimental designs
where multiple features are manipulated simultaneously are more frequently seen
in engineering and physical sciences domains, where the system units tend to be
under stricter control. They're less seen in the social and medical realms,
where individual differences can impede experiment creation and analysis.